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RELATED PATENT APPLICATION 

Represent invention describes an alternative and mWm-***,**^^ 
^^tsofquaUtyofse*^ Igth 
20 April 2000. 

BACKGROUND - FIELD OF INVENTION 

This invention relates to methods of estimating the subjective quality of a multimedia 
communications system or distributed applications software system in which the quality of the 
receivedaudio, voice, ^qMta^tw*^^^ / 

25. tme varying impairments and in which it is accordingly desirable to have some means of 
measurhgorestinutmg^ 
or software system. 

It is desirable to have some means of measuring the quality of a distributed system in order that it 
^"dthat^ 
30 the case of a speech transmission system such as a telephone network this measurement is 
sometunes made objectively by using a test group of people or test surveys. The results of a 
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subjective voice quality test are usually given as a Mean Opinion Score (MOS). As it is 
generally impractical to conduct subjective tests during the normal operation of a speech 
transmission system objective testing methods have been developed. These use measured 
statistics related to the speech transmission system to predict a subjective performance level and 
typically determine an R rating factor. 

Modem communications systems often used some form of shared transmission medium, for 
example wireless, local or wide area packet switched network. This increases the prevalence of 
time varying transmission impairments unlike the more constant and predictable impairments 
characteristic of older circuit switched networks. 

It has been found during subjective testing that bursts of noise or distortion occurring late in a 
call have greater effect than the same bursts occurring early in the call, as illustrated in Figure 3. 
This effect is thought to be due to the operation of human short-term memory which retains 
information on the most recent events and remembers detailed information on heard events for 
typically two minutes. If a series of bursts of noise or distortion occur during a telephone 
conversation then according to the known and accepted psychological theories of short term 
memory and recency the user would tend to remember the last burst and have a general 
recollection of the call quality up to that point. Experiments by Boneau and Daily 
(Psychological Association of America, 1995) indicate that the short-term memory loss process 
is exponential in nature. 

Current systems for estimating voice quality do not consider the effects of the temporal location 
of impairments on the user and hence do not give an accurate representation of subjective voice 
quality. The present invention relates to methods for estimating subjective voice quality that 
consider temporal effects and hence can provide greater accuracy. 

Distributed applications software can exhibit similar problems. Network or server loading may 
vary with time, which results in a user experiencing variable response time. It is expected that 
detailed subjective testing of user satisfaction would reveal similar characteristics to those found 
in testing telephone networks as the same basic memory processes are at work. The present 
invention provides a new approach for estimating the subjective quality of distributed 
applications software. 
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BACKGROUND - DESCRIPTION OF PRIOR ART 
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Prior art systems for measuring voice quality, as described by Douskalis (Hewlett Packard 2000), 
Royer (US 5,710,791) and Di Pietro (US 5,867,813), use centralized test equipment which 
samples the voice quality from various conversion points. A loop back condition is established 
at a conversion point, the test equipment transmits a known signal and then compares the 
received (looped back) signal with the original, thereby estimating delay, distortion and other 
impairments. This approach provides an accurate measure of voice distortion but only provides 
this measure for a sample conversion point and under the network conditions that existed at the 
time of the test. No provision is made in these systems for estimating the effects of the temporal 
location of impairments on quality. 

Another approach currently used for estimating voice quality is to estimate the subjective 
performance of the voice connection using objectively measured parameters. Models such as the 
E-Model described by Johannesson, (IEEE Communications Magazine 1997), are able to 
produce R ratings which can be correlated to user perceived voice quality. This process is 
applied by a central management system which gathers statistics on noise and delay and then 
produces an estimate of voice quality. This method as described by Johannesson does not 
consider the effects of the temporal location of impairments. 

Rosenbluth (Tl committee contribution Tl Al. 7/98-031) described the result of a subjective 
voice quality test conducted by AT&T. This test employed a set of 60 second voice samples 
which were corrupted at the beginning, middle and end. Rosenblulh reported that the MOS score 
typically ranged from 3.8 for a sample corrupted at the beginning to 3.2 for a sample corrupted at 
the end. Rosenbluth proposed a mathematical model for determining the estimated voice quality 
of the speech sample which comprised the steps of dividing the sample into segments, computing 
a MOS score for each segment, computing a weight for each segment that was a function of the 
proportional position of the segment within the overall sample and then computing the sum of 
the segment MOS scores multiplied by their respective weights. This approach suffers from 
several major drawbacks: 

(i) Rosenbluth determined the weight for each impairment based on the proportional 
location of the impairment within the speech segment - i.e. determined a number 
between 0 and 1 that represented the position of the impairment between the start 
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and the end of the segment. This is counter-intuitive as it would give the same 
weight to an impairment occurring 30 seconds before the end of a one minute call 
to an impairment occurring 30 minutes before the end of a one hour call whereas 
the process by which information is lost from human memory is time dependent. 

(ii) The equations used by Rosenbluth were computationally complex, which would 
be acceptable in an off-line determination of voice quality based on recorded data 
but not for real-time computation. 

(iii) Rosenbluth proposed a testing methodology in which a series of short test 
messages of known time duration could be sent and then the results from each test 
message combined. Rosenbluth did not consider a test methodology in which 
quality was continually estimated during the normal operation of the system. 

Accordingly, there is a need for a means of estimating subjective quality in a voice 
communications system that considers the temporal location of impairments and is 
computationally efficient. 

Furthermore, there is a need for a means of estimating subjective quality in a voice 
communications system that considers the temporal location of impairments and is able to 
continuously monitor voice quality during the normal operation of the communications system. 

Furthermore, there is a need for a means of estimating subjective quality in a video transmission 
system that considers the temporal location of impairments and is computationally efficient. 

Furthermore, there is a need for a means of estimating subjective quality in an audio transmission 
system that considers the temporal location of impairments and is computationally efficient 

Furthermore, there is a need for a means of estimating subjective quality in a distributed 
applications software or client-server system that considers the temporal location of impairments 
and is computationally efficient. 

SUMMARY 

The present invention provides an improved means of estimating subjective quality in a 
communications systems wherein the temporal location of impairments within the data stream 
are considered and the subjective effects of said impairment determined based on the time delay 
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between the cecums of said impairment and a time reference at which the quality estimate is 
being determined. 

Objects and advantages 

Accordingly, besides the objects and advantages of the present invention described above 
5 several objects and advantages of the present invention are: 

(a) to provide a method for estimating the subjective or perceptual quality of a 
multimedia communications system which considers the temporal location of 
impairments 

(b) to provide a computationally efficient method for estimating the subjectr 
perceptual quality of a multimedia communications system 

(c) to provide a computationally efficient method for estimating the subject! 
perceptual quality of a distributed applications software system 

Further objects and advantages are to provide a method of estimating subjective quality for an 
audio or video streaming system in which packetized audio or video is broadcast or multicast 
through a packet network and for a multimedia conferencing system. Still further objects and 
advantages will become apparent from consideration of the ensuing description and drawings. 

DRAWING FIGURES 

Embodiments of the invention (the Quality Monitor) are described below by way of example 
only with reference to Figs. 1 to 5 of the accompanying drawings, of which: 

Fig. 1 shows an example of the use of the Quality Monitor in a multimedia communications 
system 

Fig. 2 shows an example of the use of the Quality Monitor in a distributed software applications 
system 

Fig. 3 shows an example of the effects of temporal location on subjective voice quality 
Fig. 4 shows the architecture of the Quality Monitor 
Fig. 5 shows a flowchart for the Quality Monitor 
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Fig. 6 shows an example of the operation of the Quality Monitor. 

DESCRIPTION 

Preferred embodiment 

Fig. 1 shows an example of a multimedia communications system. Audio or video source (6) 
5 transmits a stream of audio or video data to user interface (5) by means of transmitter (1), 

transmission network (2) and receiver (3). Transmission network (2) is subject to time varying 
impairments which may include jitter, noise, distortion, variations in amplitude and other such 
impairments. Quality Monitor (7) is connected to or contained within Receiver (3), monitors the 
received data stream and reports the estimated subjective quality to management system (8). 

Fig. 2 shows an example of a distributed applications software system. A user interacts with a 
remote Server (10) by means of User Interface (13), Client Software (12) and Transmission 
Network (1 1). Requests from the user for data are submitted via User Interface (13) to Client 
Software (12). Client Software (12) forwards a representation of the user request through 
Transmission Network (1 1) to Server (10). Server (10) sends a response through Transmission 
Network (1 1) to Client Software (12). Client Software (12) communicates the response to the 
user by means of User Interface (13). The response time between the user request and the 
response may vary depending on the loading and capacity of Transmission Network (1 1) and 
Server (10). Quality Monitor (7) is connected to or contained within Client Software (12), 
monitors the transmitted and received data stream and reports the estimated subjective quality to 
management system (8). 

Fig. 3 shows some results (from Rosenbluth) of the effects of temporal location on subjective 
quality for a voice communications system, by way of illustration to support understanding of 
the following description. It can be seen that for an impairment occurring early in the call the 
subjective voice quality was 3.8, which is quite good quality, whereas the identical impairment 
25 occurring at the end of the call resulted in a subjective quality of 3.2, which is considerably 
lower quality. 

Fig 4. shows the architecture of Quality Monitor (8) as employed in a voice communications 
system. The structure of Quality Monitor (8) for other applications would be similar however 
the terminology used for impairments may vary. 
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Impairment Measurement functions (20), (21), (22) are connected to an Instantaneous Voice 
Quahty R, function (23) and measure the value of transmission system impairments during a 
time mterval. In the preferred embodiment Impairment Measurement function (20) measures 
end to end transmission delay, Impairment Measurement function (21) measures packet loss 
percentage and Impairment Measurement function (22) measures the quality of the base voice 
codmg algorithm. Instantaneous Voice Quality function (23) is connected to one input of a 
Comparator (24), an Increment function (27) and a Decrement function (26). The inputs of R 
Counter (25) are connected to Increment function (27) and Decrement function (26) The 
outputs of R Counter (25) are connected to Comparator (24), Average function (28), Minimum 
function (29), Maximum function (30) and to Output to Management System function (31) 
Average function (28) is fed back to Increment function (27). 

Fig. 5 shows a flowchart of the operation of Quality Monitor (8) as employed in a voice 
communications system. 

Fig. 6 shows an example of the output of Quality Monitor (8) as employed in a voice 
15 communications system. 

Description of Operation 

The following sequence of operations is executed at periodic intervals. In the preferred 
embodiment said periodic interval is one second however shorter or longer times may be used 
A transmission quality factor R is determined using the procedures described in detail below 
20 Subjective voice quality is related to quality factor R according to the relationship- 
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process of identifying which voice coding algorithm is being used and then using a 
predetermined table of values to determine the value Rc. A typical initial R* value for a voice 
coding algorithm is 84. 

Impairment measurement function 2 (21) determines the quality reduction due to the end to end 
5 transmission delay. Flowchart step Determine Quality Degradation (41) comprises the process 
of using the measured end to end delay to determine the delay quality degradation factor R<i. In 
the preferred embodiment this is determined using the piecewise linear approximation: 

If delay < 1 75mS then Flowchart Step 4 1 

Rd = delay * a 

10 else 

Rd = (delay -b)*c 

In the preferred embodiment delay is the one way transmission delay plus the delay due to the 
voice compression process, a = 0.02, b « 147 and c = 0.125 however other values may be used. 

Impairment measurement function 3 (22) determines the quality reduction due to the percentage 
15 of voice packets lost since the last measurement. Flowchart step Determine Quality Degradation 
(42) comprises the process of using the measured packet loss to determine the packet loss quality 
degradation R p . In the preferred embodiment this is determined using the piecewise linear 
approximation: 

If loss < 5% then Flowchart step 42 

20 Rp = loss * ei 

else 

Rp = loss * e 2 +f * (ei - ea 

In the preferred embodiment loss is the percentage of packets lost since the last measurement, ej 
25 = 4, e2 = 2.5 and f = 5 however other values may be used. Preferably the values of ei and e 2 are 
obtained from a table which holds specific values for each voice coding algorithm. 



The instantaneous voice quality factor Rj (23) is then determined by subtracting the reduction 
factors (21) and (22) from the base R value (20) for the voice coding algorithm. Flowchart step 
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R-R + (R av -R)/g flowchart step 45 

else 

R = R-h fl . 

flowchart step 46 
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20 N = N+1 
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The subjective effects of additional impairments such as jitter, clipping, noise and amplitude 
limiting may be determined and subtracted from the value of Ri using the same basic procedures 
as described above. 

5 The value of Ra V may be determined either by averaging the output value R or the instantaneous 
value Rj. 

The periodically obtained sets of measured impairment values may be stored and at some later 
time retrieved for analysis using the same procedure described above. 

An alternative means of determining a quality factor that represents the temporal position of 
1 0 impairments is to measure the impairment values at regular intervals, at each such interval to 
compute an estimated subjective quality value, and to compute a weighted sum of each 
subjective quality value multiplied by a function of the time difference between the time at 
which the impairment measurement was made and the time reference for which the subjective 
quality is being determined. 

1 5 The present invention may be used to measure the subjective quality of a distributed applications 
system, as shown in Figure 2. The measured impairments would include signal reception data 
rate and response time, both of which may be affected either by network bandwidth or server 
loading. 

Conclusion, Ramifications and Scope 

20 One benefit of the present invention is that the subjective speech quality of a voice 

communications system may be determined more accurately as the effects of the temporal 
location of impairments are considered. 

Another benefit of the present invention is that the computational complexity is low and hence 
the quality monitoring system can be implemented at low cost. 

25 Although the description above contains many specificities these should not be construed as 
limiting the scope of the invention but as merely providing illustrations of some of the presently 
preferred embodiments of this invention. For example, the present invention could be used for 
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monitoring the quality of a cellular telephone system, a digital broadcast video system or the user 
interface on a software application. 

Thus the scope of the invention should be determined by the appended claims and their legal 
equivalents, rather than by the examples given. 
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I claim: 

1 . A method for automatically estimating the subjective quality of a signal carried over a 
transmission path and presented at a user interface comprising the steps of: 
5 (a) at periodic intervals monitoring the transmission path used to carry said signal and at 
each of said monitoring intervals 
(b) determining the values of one or more impairment factors and for each set of values so 
obtained estimating the effect that said impairment factors have on the subjective or 
perceptual quality of said signal and 
10 (c) determining the estimated subjective quality of said signal by combining the estimated 
effects of each of said impairment factors and 
(d) reducing the value of a counter if said combined estimated subjective quality is below the 
value of said counter or increasing the value of said counter if said combined estimated 
subjective quality is above the value of said counter. 
15 2. A method as defined in claim 1 further characterized in that said signal is selected from the 
set of digitized voice, digitized audio and digitized video signal. 
3 . A method as defined in claim 1 further characterized in that one of said impairment factors is 
the response time between a request being sent from said user interface to a server and a 
response being received from said server. 
20 4. A method as defined in claim 2 further characterized in that said increasing in value of said 
counter is performed in increments which are a function of the difference between said 
estimated subjective quality and the average value of said estimated subjective quality. 

5. A method as defined in claim 3 further characterized in that said increasing in value of said 
counter is performed in increments which are a function of the difference between said 

25 estimated subjective quality and the average value of said estimated subjective quality. 

6. A method for automatically estimating the subjective quality of a signal carried over a 
transmission path and presented at a user interface comprising the steps of: 

(a) monitoring the transmission path used to carry said signal at periodic intervals and at 
each of said monitoring intervals determining the values of one or more transmission 
30 impairment factors and a time reference and storing said values and time references in 

sets for later processing 
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(b) retrieving said sets of stored values and for each set 

(i) estimating the effect that each member of said set has on the subjective or 
perceptual quality of said signal and 

(ii) determining the estimated subjective quality of said signal by combining the 
estimated effects of each member of said set and 

(Hi) reducing the value of a counter if said estimated subjective quality is below the 
value of said counter or increasing the value of said counter if said estimated 
subjective quality is above the value of said counter. 

7. A method as defined in claim 6 further characterized in that said signal is selected from the 
set of digitized voice, digitized audio and digitized video signal. 

8. A method as defined in claim 6 further characterized in that one of said impairment factors is 
the response time between a request being sent from said user interface to a server and a 
response being received from said server. 

9. A method as defined in claim 7 further characterized in that said increasing in value of said 
counter is performed in increments which are a function of the difference between said 
estimated subjective quality and the average value of said estimated subjective quality 

10. A method as defined in claim 8 further characterized in that said increasing in value of said 
counter is performed in increments which are a function of the difference between said 
estimated subjective quality and the average value of said estimated subjective quality. 

1 1 . A method for automatically estimating the subjective quality of a signal comprising the steps 

(a) periodically monitoring the transmission path used to carry said signal and determining 
the values of one or more transmission impairment factors and for each set of values so 
obtained estimating the effect that said impairment factors have on the subjective or 
perceptual quality of said signal and 

(b) determining the estimated subjective quality of said signal at each of said periodic 
monitoring times by combining the estimated effects of each of said impairment factors 
and 

(c) multiplying each of said estimated subjective effects by a factor which is a function of the 
time difference between said monitoring time and a reference time for which a weighted 
total subjective quality is being determined and 
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(d) determining an estimated weighted total subjective quality by summing the set of said 
subjective quality estimates. 

12. A method as defined in claim 1 1 further characterized in that said signal is selected from the 
set of digitized voice, digitized audio and digitized video signal. 

13. A method as defined in claim 1 1 further characterized in that one of said impairment factors 
is the response time between a request being sent from said user interface to a server and a 
response being received from said server. 
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Figure 1 - Multimedia communications system 
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Figure 2 - Distributed Applications Software system 
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